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(57) An image compositing apparatus of the present 
invention composites a video image and a plurality of 
graphical images' together into a final composite image. 
The image compositing apparatus Includes a first ob- 
taining unit, a first compositing unit, a calculating unit, a 
second obtaining unit, and a second compositing unit. 
The first obtaining unit obtains the graphical images and 
composition information that includes an image com- 
positing order and is used for calculating the composi- 
tion ratios of the images to the final composite image. 
The first compositing unit composites the graphical im- 
ages together into one composite graphical image ac- 
cording to the composition infomiation. The calculating 
unit calculates the composition ratio of the video image 
to the final composite image according to the composi- 
tion information. The second obtaining unit obtains 
frames of the video image. The second compositing unit 
composites the frames with the composite graphical im- 
age using the composition ratio of the video image. 
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Description 

Technical Field 

5 [0001] The present invention relates to an innage compositing apparatus for compositing a plurality of Images to 
output a final image. 

Background Art 

10 [0002] Recently digital broadcasting has started. In the digital TV set used for the digital broadcasting, graphical 
images are generated according to data from a broadcasting station and the generated graphical Images are overlaid 
on video images of a program when displayed. 

[0003] For this purpose, the digital TV set is equipped with an image compositing apparatus. 
[0004] The image compositing apparatus performs a predetermined operation between pixels for images F and G 
15 to output an image H. In the inter-pixel operation, a pixel f(x,y) of the image F and a pixel g(x,y), which corresponds 
to the pixel f(x,y), of the image G are used to generate a pixel h(x,y) of the image H. 

[0005] One method of the inter-pixel operation Is the a blending algorithm. In the a blending algorithm, the weighted 
average of the pixels f(x,y) and g(x,y) is the pixel h(x,y). Suppose that the weighting factor for the pixel g(xy) is "a", 
the a blending algorithm is given below. 

20 

h(x,y) = a*g(x,y)+(1-art(x,y) 
where, "*"lndlcates multiplication Equation Group 1 

25 

[0006] In this equation, the weighting factor is referred to as an "a value", a "transparency", a "blending factor" or 
the like. The value of the weighting factor is "0" to "1". 

[0007] When the a value is "0", the pixel g(x,y) is completely transparent and the pixel f(x,y) as It is becomes the 
combined pixel h(x,y). On the other hand, when the a value is "1 the pixel g(x,y) is completely opaque and the pixel 
g(x,y) as it is becomes the pixel h(x,y). When the a value is "0.5", the pixel h(x,y) is a composite of 50% of the pixel g 
(x,y) and 50% of the pixel f(x,y). In this manner, the a value indicates how an Image Is overlaid on another image. In 
the actual hardware, each pixel is represented by RGB (Red, Green, and Blue) color components, so that the arithmetic 
operation indicated above Is performed for each of the components. 

[0008] For compositing graphical images and video images together in real time, the image compositing apparatus 
35 includes an OSD (On Screen Display) plane, which is a memory area for developing graphical images, and a video 

plane, which is another memory area for developing video images In frame units. Whenever a frame on the video plane 

is updated, images in the two areas are composited together according to the a blending algorithm to output a composite 

image. Fig. 1A is a conceptual diagram showing how images on an OSD plane 2501 and a video plane 2502 are 

composited together to output a composite image 2503. 
40 [0009] Meanwhile, the recent digital broadcasting requires to overlay a plurality of graphical images 2511, 2513, 

2514, and 2515, which Indicate the title and TV programs and the like, on a video image 2512 of a program while the 

video image 2512 Is reproduced as shown In Fig. 18. 

[0010] Theoretically, it is enough to perform the a blending' algorithm for images to be composited together in turn 
from the bottom to the top in order to composite a video image and a plurality of graphical images together. Actually, 
45 however, a large amount of arithmetic operations is performed for the a blending algorithm for a plurality of images, 
so that it is extremely difficult to composite a plurality of Images in real time whenever a frame of video images is updated. 
[001 1 ] Images can be composited at high speed by providing the planes that correspond to the Images and a plurality 
of pieces of hardware that perform the a blending algorithm and that also correspond to the images. In this case, 
however, hardware cost is raised. Also, many planes are needed. This is problematic. 

50 

Disclosure of the Invention 

[001 2] It is accordingly an object of the present invention to provide an image compositing apparatus that composites 
images together in real time according to the reproduction rate of video images, i.e., at high speed. 
55 [0013] Another object of the present Invention Is to provide an Image compositing apparatus with fewer memories 
for image development. 

[0014] The above-mentioned first and second objects may be achieved by an image compositing apparatus that 
composites images together Into a final composite image, the images including a video image and a plurality of graphical 
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images, the image compositing apparatus including: a first obtaining unit that obtains the graphical images and com- 
position information including an image compositing order, the composition information being used for calculating a' 
composition ratio of each of the Images to the final composite image; a first compositing unit that composites the 
graphical images to generate one composite graphical image according to the composition infomiation; a calculating 
5 unit that calculares a composition ratio of the video image to the final composite Image according to the composition 
information; a second obtaining unit that obtains frames of the video image; and a second compositing unit that com- 
posites one of tie obtained frames at a time with the composite graphical Image using the composition ratio of the 
video image. 

[0015] In this image compositing apparatus, the graphical images are composited together and the composition ratio 

10 of the video imege is calculated before the frames of the video image are composited with the composite graphical 
image. Accordingly, it is not necessary to composite a plurality of graphical images with a video image for each of the 
frames unlike the conventional manner. As a result, the calculation Is performed under light load and the processing 
speed is increased Due to the higher processing speed, images are composited and a final composite image is dis- 
played in real time according to the reproduction rate of the video image. Accordingly, the capacity of the frame buffer 

15 for developing video image frames can be decreased. This Is effective. 

[0016] The first and second objects may be also achieved by the composition information further including: coeffi- 
cients, each of which corresponds to a different one of the images; and arithmetic infomriation which has blending 
algorithms which each correspond to the images, the blending algorithms using the coefficients. 
[0017] In this mage compositing apparatus, a plurality of graphical images and a video image can be composited 

20 together for which the coefficients and arithmetic information have been defined. This is also effective. 

[0018] The first and second objects may be also achieved by a structure in which the image compositing apparatus 
further includes a first frame buffer that stores images , and a second frame buffer that stores frames of the video image, 
wherein the first compositing unit reads the graphical images obtained by the first obtaining unit, according to the image 
compositing order, composites each of the read graphical images with a storage content of the first frame buffer using 

25 the coefficients and the arithmetic information, and replaces the storage content of the first buffer with a result of 
compositing one of the read. graphical images with the storage content, the second obtaining unit stores the obtained 
frames in the second buffer and the second compositing unit composites each of the frames stored in the second 
buffer with an image in the first frame buffer using the composition ratio of the video image. 

[0019] In this image compositing apparatus, a plurality of graphical images and video image can be composited 

30 together only wl:h two frame buffers, i.e.. the first and second frame buffers. 

[0020] The first and second objects may be also achieved by the image compositing having a structure in which the 
first compositing unit performs a blending algorithm on the image In the first frame buffer using a coefficient and arith- 
metic Infomnation corresponding to the video Image after compositing a graphical image Immediately preceding the 
video Image In the compositing order and before compositing a graphical image' immediately succeeding the video 

35 image with the s o rage content, and replaces the content of the first frame buffer with a result of the blending algorithm. , 
[0021] In this image compositing apparatus, a plurality of graphical Images and video Image can be composited 
together with accurate composition ratios even If the video image is positioned between graphical Images. 
[0022] The fir3t and second objects may be also achieved by the image compositing further including a display 
screen, wherein the first compositing unit generates the composite graphical image, the second obtaining unit obtains 

40 the frames, and the second compositing unit composites the frames with the composite graphical Image in parallel 
with each other 

[0023] In this image compositing apparatus, a state of the images in the compositing process is displayed on the 
display screen. As a result, a situation In which nothing Is displayed on the display screen until the completion of the 
image composit ng can be prevented. 
45 [0024] The first and second objects may be also achieved by the composition infomiation further including a com- 
position coefficient for each of the images, each composition coefficient Indicating a composition ratio of one of the 
images to a composite of the image with at least another one of the images. 

[0025] In this mage compositing apparatus, a plurality of graphical images and a video image can be composited 
together for which the compositing order and composition ratios to the composite of any images among the images 
50 have been defined. 

[0026] The first and second objects may be also achieved by a structure in which the image compositing order 
indicates a first overlaying order of the images, the composition coefficients are a values, each of which indicates a 
composition ratio of one of the images to a composite of the image with images below the image in the first overlaying 
order, and the calculating unit calculates the composition ratio of the video image to the final composite image from a 
55 values of the viceo image and images above the video image in the first overlaying order 

[0027] In this inage compositing apparatus, still and video Images can be composited together for which the a values 
and the compositing order have been defined. The layout of images for which the a values have been defined can be 
more flexibly changed compared with images for which the composition ratios to the final composite image have been 
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defined. This is an advantage. Accordingly, the Image connpositing apparatus that composites images using a values 
can deal with a variety of EPG display layouts. 

[0028] The first and second objects may be also achieved by the image compositing apparatus further Including a 
switching unit for switching two adjacent images in the first overlaying order to rearrange the images in a second 

5 overlaying order; and an updating unit for updating a values of the switched two adjacent images so that results of 
compositing in the first and second overlaying orders are the same, wherein the first compositing unit generates the 
composite graphical image, the calculating unit calculates the composition ratio of the video image to the final composite 
image, and the second compositing unit composites the frames with the composite graphical image using the second 
overlaying order and the updated a values. 

10 [0029] In this image compositing apparatus, a plurality of images can be composited together even if the order of 
the Image is changed. 

[0030] Also, when the order is changed so that a video image in the middle of the order is positioned at the uppermost 
in the image compositing apparatus, the plurality of graphical images can be composited together in turn in the bottom- 
to-top order and the video image can be composited with the composite graphical image. Accordingly, the amount of 

15 calculation Is decreased and the calculation is perfomied under tight load. 

[0031] The first and second objects may be also achieved by the image compositing apparatus having a structure 
which further includes a storage unit for storing a plurality of graphical images that are obtained by the first obtaining 
unit, wherein the graphical Images are each made up of (A) image data which has pixels, a number of which Is no 
larger than pixels of the final composite image and (B) layout information which Indicates a layout of the images on 

20 the final composite Image, and the first compositing unit generates the composite graphical Image, the calculating unit 
calculates the composition ratio of the video image to the final composite image, and the second compositing unit 
composites the frames with the composite graphical Image for an overlapping part of the images that is determined 
by the layout infomnation. 

[0032] The first and second objects may be also achieved by the image compositing apparatus having a structure 
25 which further comprises a storage unit for storing the plurality of graphical images obtained by the first obtaining unit; 
the plurality of graphical images are represented by vector data; and the first compositing unit generates the composite 
graphical image after converting the vector data to the pixels. 

[0033] In this image compositing apparatus, the data of graphical images are vector data, which has less amount of 
data than pixel data. Accordingly, the memory capacity can be further decreased. 

30 [0034] The first and second objects may be also achieved by an image compositing apparatus that composites 
images together into a final composite Image, the images including a plurality of video images and a plurality of graphical 
images, the image compositing apparatus including: a first obtaining unit that obtains the graphical images and com- 
position information including an image compositing order, the composition information being used for calculating a 
composition ratio of each of the images to the final composite image; a first compositing unit that composites the 

35 graphical images to generate one composite graphical image according to the composition Infomnation; a calculating 
unit that calculates composition ratios of the video images to the final composite image according to the composition 
infomiation; a second obtaining unit that obtains frames of the video images one frame at a time; and a second com- 
positing unit that composites, each time one frame is obtained, the obtained frame with the composite graphical image 
using the composition ratios of the video images. 

40 [0035] In this image compositing apparatus, a plurality of video Images and a plurality of graphical images can be 
composited together with less amount of calculation and the calculation can be performed under light load. 
[0036] The first and second objects may be also achieved by an image compositing apparatus that composites 
images together into a final composite image, the images including a video' image and a plurality of graphical images, 
the image composition apparatus including: a first obtaining unit that obtains the graphical images; a first compositing 

45 unit that composites the graphical images to generate one composite graphical image; a second obtaining unit that 
obtains frames of the video image; and a second compositing unit that composites one of the obtained frames at a 
time with the composite graphical image. 

[0037] It is not necessary for this image compositing apparatus to include the same number of memories as the 
number of graphical images that have the same number of pixels as the display screen. Accordingly, the memory 
50 capacity can be decreased. 

[0038] Also, a plurality of graphical Images are composited together before compositing a video image with the 
composite graphical image. Accordingly, it is not necessary to composite graphical images with a video image for each 
of the frames. As a result, the amount of calculation can be decreased and the calculation can be performed under 
light load. 

55 
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Brief Description of the Drawings 
[0039] 

5 Fig. 1 A is a conceptual diagram showing how images on an OSD plane 2501 and a video plane 2502 are com- 

posited together to output a composite image 2503; 

Fig. 1B shows how still and video image components are overlaid on each another; 

Fig. 2 is a bock diagram showing the structure of an image compositing apparatus according to the first embodiment 

of the present invention; 
*o Fig. 3A shows a composite image 201 ; 

Fig. SB shows components of the composite Image 201 ; 

Fig. 4 is a diagram for explaining the structure of an image file; 

Fig. 5 shows an example of the Index file; 

Fig. 6 shows the structure of video Image data; 
^5 Fig. 7 shows the pixel data In video image data 501 In the visual form; 

Fig. 8 shows the data structure of first composite image data; 

Fig. 9 is a flowchart showing the first compositing processing; 

Fig. 1 0 Is a flowchart showing the second compositing processing; 

Fig. 1 1 is a chart for explaining the flow of the operations by Image compositing apparatus 1 00 at the time of EPG 
20 display; 

Fig. 12 shows a component group, which is composed of N+1 video and/or graphical image components; 

Fig. 13 shows a program for the arithmetic operations in the first embodiment of the present Invention written in 

the form of the C language; 

Fig. 14 shows a modification of the program In Fig. 13; 

Fig. 15 is a block diagram showing the structure of an Image compositing apparatus according to the second 

embodiment of the present Invention; 

Fig. 1 6 is a flowchart showing fourth compositing processing; 

Fig. 17 is a chart for explaining the flow of the operations by an Image compositing apparatus 200 at the time of 

EPG display; 

30 Fig. 1 8 Is a diagram for explaining how to switch two adjacent components in the overlaying order of components; 

Fig. 1 9 shews how to switch video and graphical image components so that the video Image component is posi- 
tioned at the uppermost; 

Fig. 20 Is a block diagram showing the structure of an image compositing apparatus according to the third embod- 
iment of the present invention; 
35 Fig. 21 Is a flowchart showing seventh compositing processing; 

Fig. 22 is a block diagram showing the structure of an image compositing apparatus according to the fourth em- 
bodiment of the present invention; 
Fig. 23A shows an example of index file; 

Fig. 238 shows the correspondence between the number and the operation type; 
40 Fig. 24 is a flowchart showing ninth compositing processing; 

Fig. 25 shows Porter-Duff rules according to the operation types; 

Fig. 26 shows Porter-Duff rules according to the operation types; 

Fig. 27 shows Porter-Duff rules according to the operation types; 

Fig. 28 is a flowchart showing tenth compositing processing; 
45 Fig. 29 shows Porter-Duff rules according to the operation types; 

Fig. 30 Porter-Duff rules according to the operation' types; and 

Fig. 31 shows a program for the arithmetic operations In the fourth embodiment of the present Invention written in 
the form of the C language. 

50 Best Mode For Carrying Out the Invention 

[0040] Explanations of preferred embodiments of the present Invention will be given below with reference to figures. 

<The First Embodiment> 

<Structure> 

[0041] Fig. 2 Is a block diagram showing the structure of an Image compositing apparatus according to the first 
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embodiment of the present invention. 

[0042] In Fig. 2, an image compositing apparatus 1 00 includes an input unit 1 01 , a controller 1 02, an EPG generating 

unit 1 03, an image holding unit 1 04, a video reproducing unit 1 05, a video plane 1 06, an OSD plane 1 07, a first com- 
positing unit 108, a second compositing unit 109, a third compositing unit 110, and an output unit 111 . 
5 [0043] The input unit 101 includes a remote controller and front panel buttons. Using the remote controller and the 
front panel buttons, the user inputs instructions. More specifically, the user inputs instructions to turn the image com- 
positing apparatus 100 ON and OFF, to switch channels, and to turn the EPG (Electronic Program Guide) display ON 
and OFF 

[0044] Here, the "EPG" indicates a system for displaying program tables, information on the program contents, and 
10 the like on the TV screen. Also, the "EPG display" indicates the condition in which a program table, for instance, is 
displayed on the' TV screen. The EPG can be applied to searching for programs on the EPG display screen by a genre 
or a cast. Also, an EPG -compatible VCR can be easily programmed to record. 

[0045] When the EPG display is ON, the image compositing apparatus 100 displays a composite image 201 shown 
in Fig. 3A, for instance. 

15 [0046] The composite image 201 in Fig. 3A Is composed of components 202, 203, and 204 as shown in Fig. SB. The 
component 202 is a graphical image indicating a program table, the component 203 is a video image indicating the 
content of a program of one channel, and the component 204 is a graphical image indicating the title of the program 
shown by the component 203. The components 202 and 204 are generated according to the EPG information from 
■ the broadcasting station. 

20 [0047] As shown in Figs. 3A and 3B, the image compositing apparatus 100 displays the component 202 while the 
EPG display is ON. Also, the component 203 is displayed on the bottom right and the component 204 on the upper 
left of the component 202. The frame rate of a video image of the component 203 is dozens of frames per second. 
[0048] As has been described, the image compositing apparatus 1 00 has a function of generating and displaying a 
composite image from a plurality of components including a plurality of graphical images and one video image. 

25 [0049] Here, explanations will be given using Figs. 3A and 3B as necessary. Note that when not needed to be dis- 
tinguished from each other, the still and video image components are referred to as just the "component". 
[0050] The controller 1 02 controls all the elements of the image' compositing apparatus 1 00. More specifically, when 
the user instructs to turn the image compositing apparatus 1 00 ON, the controller 1 02 has the video reproducing unit 
105 reproduce a program. When the user instructs to switch channels, the controller 102 has the video reproducing 

30 unit 1 05 reproduce another program. When the user instructs to turn the EPG display ON, the controller controls the 
EPG generating unit 103, the video reproducing unit 105, the first compositing unit 108, the second compositing unit 
109, and the third compositing unit 11 0 so as to generate the composite image 201 . The processing relating to the 
EPG display will be described later. 

[0051] The EPG generating unit 1 03 obtains and holds the EPG infonnation from the broadcasting station. The EPG 
35 information includes layout Infonnation on the sizes, positions, and overlaying order of components and graphical 
image information on graphical image components. The graphical image information, which corresponds to the image 
contents of graphical image components, includes texts, graphics data and the like. The EPG generating unit 103 
generates a plurality of image files and one index file according to the EPG information and stores the Image files and 
index file in the image holding unit 104. Then, the EPG generating unit 103 extracts from the EPG infonnation the size 
40 and position of the video image component that is a part of the EPG display screen. The extracted size and position 
are output to the video reproducing unit 105. 

[0052] The image files, which correspond to the components 202, 203, and 204, in Fig. 3B, includes the sizes, 
positions and image data of the components. For the graphical image component, the image data is generated by the 
EPG generating unit 103 according to the graphical image information. 

45 [0053] The index file, which Is used for managing the plurality of image files, includes a plurality of pieces of image 
information. Each piece of image information corresponds to a different Image file. The plurality of pieces of image 
information are arranged In the index file in the same order as the overlaying order of components in Fig. 3A. Each 
piece of image information includes an image type and a storage position. The image type indicates whether the image 
of the corresponding image file is a graphical image or a video image. The storage position indicates the position of 

50 the beginning of the corresponding image file In the image holding unit 1 04. 

[0054] Here, an explanation of a specific example of the image file and index file will be given with reference to Figs. 
4 and 5. 

[0055] Fig. 4 is a diagram for explaining the structure of the image file. 

[0056] In an image file 301 in Fig. 4, the first line indicates the position, the second line the size, and the third and 
55 the following lines the image data. The image data is composed of a group of pixel data that correspond to the pixels 
of the component. Each piece of the pixel data includes RGB components that indicate the color of the pixel and an a 
value that indicates the transparency of the pixel. Each a value indicates a composition ratio of one of the images to 
a composite of the image with images below the image in the overlaying order of images. A storage position 302 at 
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the left of the image file 301 indicates the storage position of each piece of the pixel data with reference to the position 
of the beginning of the image file 301 , "0000". 

[0057] In the image file 301 , the third line indicates the pixel data corresponding to the pixel with the coordinates 
(0,0), the fourth line the pixel data corresponding to the pixel with the coordinates (1 ,0), and the fifth line the pixel data 
5 corresponding to the pixel with the coordinates (2. 0). Here, the coordinates (0,0) are located at the upper left corner 
of the component. In this way, each piece of the pixel data corresponds to a different pixel in the left-to-righl and lop- 
to-bottom order. 

[0058] The position indicates the coordinates of the component in the composite image 201 . The position is repre- 
sented by the x and y coordinates of the upper left corner of the component by talking the upper left corner of the 
10 composite imago 201 as the origin. 

[0059] The size Indicates the height "H" and the width "W" of the rectangular component. The height "H" and width 
"W" are each represented in pixel units. 

[0060] Each of the RGB components is represented by a value from "0" to "265". When the three components are 
all "0", the pixel ::olor is black and when "255", white, 
15 [0061] The a value, which is represented by a value from "0" to "255", indicates the transparency when the pixel is 
overlaid on another pixel, i.e., how degree the lower pixel passes through the upper pixel. 

[0062] More specifically when a pixel "A" is combined with a pixel "B" with the a value, a combined pixel "C" is (cxA+ 
(255-a)B)/255. I"lere, the a value is from "0" to "255". When the a value is "0", the pixel "A" is transparent, i.e., 1 00% 
of the pixel "B" passes through the pixel "A". On the other hand, when the a value is "255", the pixel "A" is opaque, i. 
20 e., the pixel "B" does not pass through the pixel "A". In this case, the combined pixel "C" is the same as the pixel "A". 
Meanwhile, when the a value is "128", 50% of the pixel "B" passes through the pixel "A", i.e., the combined pixel "C" 
has the color that is a result of combining 50% of the pixel "A" with 50% of the pixel "B". 

[0063] Actually, the pixels "A", "B", and "C" are represented by the RGB components. The compositing results are 
calculated for each of the components. 

25 [0064] The EPG generating unit 1 03 sets the a values of the image file of each of the components 202, 203, and 
204 as follows. "I'he EPG generating unit 1 03 sets the a values of all the pixels of the component 202 at "255". This is 
because the component 202 Is positioned at the lowest layer in the composite image 201 . Also, the EPG generating 
unit 103 sets the a values of all the pixels of the component 203 at "192". For the component 204, the EPG generating 
unit 103 sets iho a values of the pixels corresponding to the characters at "255" and the a values of the remaining 

30 pixels at "64". 

[0065] Note that how to determine the RGB component values and the a values Is well known in the art and does 
not characterize the present invention, so that no more detailed explanation will be given here. 
[0066] A component 303 in Fig. 4 is a diagram for showing the correspondence between the image data and the 
component. As shown in Fig. 4, each piece of the pixel data corresponds to a different pixel in the left-to-right and top- 
35 to-bottom order. 

[0067] As has been described, the EPG generating unit 103 generates image files. Note that when generating an 
image file corresponding to a video Image component, the generating unit 103 sets all the RGB components at "0". 
More specifically, when generating an image file for a video image component, the generating unit 103 determines the 
position, size, and the a values but no RGB components. This is because the RGB components are not determined 
40 when an image tile is generated since video image components correspond to a program and are transmitted from the 
broadcasting station in real time. The RGB components of the image file corresponding to a video image component 
are detennined by the video reproducing unit 105. which will be described later. 

[0068] Fig. 5 shows an example of the index file. An index file 410 in Fig. 5 includes image infonnation 421, 422, 
and 423. The imiige information 422 is overlaid on the image information 421 , and the image information 423 is overlaid 

45 on the image information 422. More specifically, the image information 421 corresponds to the component 202 that Is 
positioned at the lowest layer, the image information 422 corresponds to the component 203 at the next layer, and the 
image information 423 corresponds to the component 204 at the uppermost layer. The values In a column 411 indicate 
the image types of the components. The value "0" indicates the graphical image, while the value "1" the video image. 
On the other hand, the values in a column 412 indicates the storage positions. 

50 [0069] Meanwhile, images 430, 440, and 450 correspond to the components 202, 203, and 204, respectively. The 
images 430, 440 and 450 show the image data In the corresponding image files In the form of image. Here, the image 
440 shows that all the RGB components of the image data corresponding to the component 203 are "0". 
[0070] The image holding unit 104 is composed of a memory, a hard disk, or the lil<e. The image holding unit 104 
holds the image file and index file that have been generated by the EPG generating unit 103. 

55 [0071] The viceo reproducing unit 1 05 receives the program from the broadcasting station and decodes the program. 
Then, the video reproducing unit 1 05 reproduces video images at the frame rate of dozens of frames per second and 
stores the video Image data on the video plane 1 06 In order. When storing the video image data, the video reproducing 
unit 1 05 determines the layout with reference to the composite Image 201 according to the position and size that have 
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been Input from the EPG generating unit 1 03. By doing so, the video reproducing unit 1 05 stores the RGB components 
in an area on the video plane 1 06 according to the layout. 

[0072] The video plane 106 Is composed of memory or the like. The video plane 106 holds video Image data that 
has been stored by the video reproducing unit 105 and a composite a value that has been stored by the second 
5 compositing unit 109. A composite a value indicates the transparency of one of a plurality of pixels when the plurality 
of pixels are combined. More specifically, the composite a values held by the video plane 1 06 indicate the transparency 
of the pixels of the video image component when the plurality of components are composited. 

[0073] Fig. 6 shows the structure of a video Image file. As shown in Fig. 6, video image data 501 is composed of a 
group of pixel data. Each piece of the pixel data is composed of RGB components and a composite a value. An image 
10 502 is a diagram for showing the correspondence between each of the pixel data and a different pixel position in the 
imago. 

[0074] Fig. 7 shows the pixel data in the video image data 501 in the visual form. As shown in Fig. 7, an image 600 
corresponding to the video image data 501 is composed of an area 601 corresponding to the component 203 and an 
area 602 corresponding to the remaining part. The image 600 has the same height and width as the composite image 

15 201 . The video plane 1 06 receives the RGB components of the area 601 from the video reproducing unit 1 05 and the 
a values from the second compositing unit 109. Meanwhile, the video plane 106 holds "0" as the RGB components 
and the a values of the area 602 in advance. The RGB components of the area 601 are updated at the rate of dozens 
of frames per second in accordance with the reproduction rate of the video reproducing unit 105. 
[0075] The OSD plane 1 07 is composed of a memory or the like. The OSD plane 1 07 holds first composite image 

20 data that has been output from the first compositing unit 1 08. 

[0076] Fig. 8 shows the data structure of first composite image data 701 . The first composite image data 701 is 
composed of a group of RGB components. The first composite image data 701 corresponds to the result of compositing 
the components 202 and 204, I.e., the compositing result of the graphical image components. The compositing will be 
described later. 

25 [0077] An image 702 is a diagram for showing the correspondence between each set of the RGB components and 
a different pixel position in the image. As shown in Fig. 8, each set the RGB components corresponds to a different 
pixel in the left-to-right and top-to-bottom order. The image 702 has the same height and width as the composite image 
201 in pixel units. 

[0078] The first compositing unit 1 08 combines the image data of the plurality of image flies that have been stored 
30 in the image holding unit 1 04 to generate the first composite image data. The first composite image data is stored on 
the OSD plane 1 07. This processing is referred to as the "first compositing processing". 
[0079] Fig. 9 is a flowchart showing the first compositing processing. 

[0080] To begin with, the first compositing unit 108 initializes the OSD plane 107 (step S800). More specifically, all 
the areas for RGB components are set to be "0" on the OSD plane 1 07. 
35 [0081] Next, the first compositing unit 108 repeats the operations at steps SB01 to 8807 to composite graphical 
image components In turn in the bottom-to-top order. 

[0082] At step S801 , the first compositing unit 1 08 reads image information "i" from the index file in the image holding 
unit 1 04. Here, the letter "i" is a variable that indicates a number assigned to a component. Numbers are assigned to 
the overlaid components from the bottom to the top in the ascending numeric order. The image information and the 
40 image file corresponding to a component "i" are referred to as the "image information "i"" and the "image file "i"". In 
this flowchart, the initial value of "i" is set to "0" and the increment is "1 

[0083] Then, the first compositing unit 108 reads from the image holding unit 104 the image file "i" that has been 
stored in the storage position indicated by the image information "i" (step S802). 

[0084] The first compositing unit 108 reads the size and the position to detennine an area on the OSD plane 107 
45 where the component "i" overlaps another component (step S803). 

[0085] The first compositing unit 1 08 judges whether the image type indicated by the image infomiation "i" Is video 
image or graphical Image (step S804). 

[0086] When the image type is graphical image, the first compositing unit 1 08 performs the a blending algorithm for 
the RGB components of the image file "i" and the RGB components in the overlapping area on the OSD plane 107 
50 (step S805). The a blending algorithm is given below. 



R(x.y) 



ai(x,y)*Ri(x,y)+(1 



ai(x,y))*R(x,y) 



55 



G(x,y) 



ai(x,y)*Gi(x,y)+(1 



ai(x,y))*G(x,y) 



B(x,y) 



ai(x,y)*Bi(x,y)+(1 



ai(x,y)rB(x.y) 



Equation Group 2 
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[0087] In thesfj equations, the terms R(x,y), G(x,y), and B(x,y) on the left sides indicate RGB components that are 
to be newly obtained. The tenns Rl(x,y), Gi(x,y), Bi(x,y), and al(x,y) indicate RGB components and an a value of the 
image file "I". On the other hand, the tenns R(x,y), G(x,y), and B (x,y) on the right sides indicate RGB components that 
are held in the overlapping area on the OSD plane 107. More specifically, the value R(x,y) on the left side is obtained 

5 as follows. The values Ri(x,y) and R(x,y) are weighted with values ai and 1 -ai, respectively. The sum of the weighting 
results is the value R(x,y) on the left side. The values G(x,y) and B(x,y) are obtained in the same manner. The first 
compositing unit 108 stores the newly obtained RGB components R(x,y), G(x,y), and B(x,y) on the OSD plane 107. 
[0088] On the other hand, when it is judged that the image type is video image at step S804, the first compositing 
unit 1 08 performs arithmetic operations according to the equations given below using the a values of the image file "i" 

10 and the RGB components in the overlapping area on the OSD plane 1 07 (step S806). 

R(x.y) = (1-ai(x,y))*R(x,y) 
15 G(x,y) = (1.ai(x,y)rG(x.y) 

B(x,y) = (i-al(x,y))*B(x,y) Equation Group 3 

[0089] In thestj equations, the tenns R(x,y), G(x,y), and B(x,y) on the left sides indicate RGB components that are 
20 to be newly obtained. The tenn ai(x,y) Indicates an a value of the image file "i". On the other hand, the terms R(x y), 
G(x,y), and B(x, y) on the right sides indicate RGB components that are held in the overlapping area on the OSD plane 
107. More specifically, the value R(x,y) on the left side is obtained by weighting the R (x, y) on the right'side with a 
value 1-ai(x,y). The values G(x,y) and B(x,y) are obtained in the same manner. 

[0090] When tne first terms on the hght sides of the equations in Equation Group 2 are set as "0", the equations in 
25 Equation Groups 2 and 3 become the same. 

[0091] In the first terms on the right sides of the equations at step S805, the RGB components of the image file are 
weighted with the value a. Accordingly, the RGB components of the video image are not weighted with the value a in 
the equations at step 8806. 

[0092] The first compositing un it 1 08 stores the RGB components R(x,y) , G(x,y) , and B(x,y) that have been calcu lated 
30 at step 8805 or S806 on the OSD plane 1 07. 

[0093] When the first compositing unit 1 08 has perfomned the processing for all the image files as has been described, 
the first compositing processing Is completed (step 8807). 

[0094] As a result of the first compositing processing, the result of the compositing the graphical image components 
is held on the OSD plane 107. 

35 [0095] The second compositing unit 109 calculates the composite a values for the video image component and 
stores the calculated composite a values to perfomn "second compositing processing". 
[0096] Fig. 10 Is a flowchart showing the second compositing processing. 

[0097] To begin with, the second compositing unit 1 09 initializes the composite a values on the video plane 1 06 (step 
S900). More specifically, all the areas for composite a values are set to be "0" on the video plane 106. The composite 

40 a values, as a result of the processing in Fig. 1 0, retain the a composite values for the video image component. 
[0098] Next, the second compositing unit 1 09 repeats the operations at steps 8901 to 8907. 
[0099] At step 8901, the second compositing unit 109 reads Image infomriation "i" from the index file in the image 
holding unit 104. Here, the letter "i" is a variable that indicates a number assigned to a component. Numbers are 
assigned to the overiaid components from the bottom to the top in the ascending numeric order. The image information 

45 and the image file corresponding to a component "i" are referred to as the "image infomnation "i"" and the "Image file 
"i"". In this flowcnart, the initial value of "i" is set to "0" and the increment Is "1". 

[0100] Then, tie second compositing unit 109 reads from the' Image holding unit 104 the image file "I" that has been 
stored in the storage position indicated by the image information "i" (step 8902). 

[0101] The second compositing unit 109 reads the size and the position to determine an area on the video plane 
50 106 where the component "i" overiaps another component (step S903). 

[0102] The second compositing unit 109 judges whether the image type indicated by the image information "i" is 
video image or graphical image (step 8904). 

[0103] When the image type is graphical image, the second compositing unit 109 calculates newly composited a 
value based on the composited a value of the image file "i" and the composited a value in the overlapping area on the 
55 video plane 106 (step 8905). The equation for this calculation Is given below. 

«(x,y) = (1-ai (x,y))*a(x,y) Equation Group 4 
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[0104] In this equation, the term a(x,y) on the left side indicates a composite a value that is to be newly obtained. 
The term al(x,y) indicates an a value of the image file "i". On the other hand, the tenn a(x,y) on the right side indicates 
an a value that Is held in the overlapping area on the video plane 1 06. 

[01 05] The second compositing unit 1 09 stores the newly obtained composite a values a(x,y) on the video plane 1 06. 
[0106] On the other hand, when it is judged that the image type is video image at step S904, the second compositing 
unit 109 perfonns arithmetic operations according to the equation given below (step S906). 



a(x,y) = ai(x,y) Equation Group 5 

10 

[0107] In this equation, the term a(x,y) on the left side indicates an a value that is to be newly obtained. The term 
ai(x,y) on the right side indicates an a value of the image file "I". More specifically, the a value ai (x, y) is the a value 
that is to be newly obtained. 

[0108] The second compositing unit 1 09 stores the a values a(x,y) that have been calculated at step S905 or S906 
15 on the video plane 106. 

[0109] When the second compositing unit 109 has performed the processing for alt the Image files as has been 
described, the second compositing processing Is completed (step S907). As a result, the composite a values are stored 
on the video plane 1 06. 

[01 1 0] When receiving normal reproduction instructions from the controller 1 02, the third compositing unit 1 1 0 outputs 
20 to the output unit 1 1 1 the RGB components that have been held on the video plane 1 06. Also, when receiving the EPG 
display Instructions from the controller 102, the third compositing unit 110 perfonns "third compositing processing" to 
composite the RGB components stored on the video plane 1 06 with the RGB components of the first composite image 
data stored on the OSD plane 107. Then, the third compositing unit 110 outputs the composite Image to the output 
unit 111. 

25 [0111] The third compositing processing is indicated by the equations given below. 

R(x,y) = a(x,y)*Rv(x,y)+Ro(x,y) 
30 G(x.y) =^ a(x,y)*Gv(x,y)+Go(x,y) 

B(x,y) = a(x,y)*Bv(x,y)+Bo(x,y) Equation Group 6 

[0112] Here, the terms R(x,y), G(x,y), and B(x,y) indicate the RGB components of each of the pixels that are to be 
35 output to the output unltlH as a result of the third compositing processing. The terms a(x,y), Rv(x,y), Gv(x,y), and Bv 
(x,y) indicate each of the composite a values and each set of the RGB components of the video image data that Is 
stored on the video plane 106. Meanwhile, terms Ro(x y), Go{x,y), andBc(x,y) indicate eachsetof the RGB components 
of the first composite image data. 

[0113] Accordingly, the third compositing unit 110 calculates the sum of RGB components of the first composite 
40 image data held on the OSD plane 1 07 and the product of RGB components and a composite a value of the video 
image data held on'the video plane 106 for each frame in the video reproduction. 

[0114] The output unit 111 Is composed of a CRT (Cathode-Ray Tube) or the like. The output unit 111 receives the 
RGB components that have been output from the third compositing unit 110 and displays the received RGB compo- 
nents. (Operations) 

45 [0115] Here, an explanation of the operations by the image compositing apparatus 100 will be given below. 

[01 16] Fig. 1 1 is a chart for explaining the flow of the operations by the image compositing apparatus 1 00 at the time 
of the EPG display. 

[0117] In Fig. 11 , operations by elements of the Image compositing apparatus 100 are Indicated in the rectangles 
and the data flow by the arrows. 

50 [01 18] When received by the Input unit 1 01 (step SI 001 ), the EPG display Instructions is sent to the other elements 
via the controller 102. When receiving the EPG instructions, the EPG generating unit 103 generates Image files and 
an Index file according to the EPG information that has been broadcast. The generated image files and index file are 
stored in the image holding unit 1 04. Also, the EPG generating unit 1 03 infonns the video reproducing unit 1 05 of the 
position and size of the video image component that has been obtained when generating the Image files (step SI 002). 

55 [0119] The first compositing unit 108 combines the RGB components In the overlapping areas In graphical image 
components stored in the image holding unit 1 04. The combined RGB components are stored on the OSD plane 1 07 
(step SI 003). 

[01 20] The second compositing unit 1 09 calculates the composite a values for the video image component according 
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to the a values of the image files that have been stored in the Innage holding unit 104. The calculated composite a 
values are stored on the video plane 106 (step SI 004). 

[0121] The video reproducing unit 1 05 reproduces the video image and stores the video image data on the video 
plane 1 06 according to the layout indicated by the size and position that have been obtained from the EPG generating 
5 unit 103 (step S1005). 

[0122] The third compositing unit 110 weighted the RGB components of the video image component with the com- 
posite a values. Then, the RGB components of the first composite Image data are added to the weighted RGB com- 
ponents to obtain the RGB components of the composite image. The obtained RGB components of the composite 
image are output to the output unit 111 (step SI 006). 
10 [01 23] The output unit 1 1 1 displays the composite image RGB components from the third compositing unit 1 1 0 (step 
SI 007). 

[0124] In Fig. 9, operations at steps SI 003, 1004, 1005, and 1006 are performed in parallel. The operation at step 
SI 006 is performed at the same rate as the reproduction rate of the video image data at step SI 005. 

15 <Supplemental Remarks> 

[0125] Suppksmental remarks will be given below on the compositing of video and graphical images, which is an 
important feature of the present Invention. 

[0126] Fig. 12 shows a component group, which is composed of N+1 video and/or graphical image components. 
20 Each of the components has RGB components and an a value an shown in Fig. 1 2. Note that the a value is aO = 1 .0 
for the component positioned at the lowest layer of the layer. The RGB components of the composite image of the 
component grojp 1101 is obtained according to the equations given below. 



25 R = Z(pi*Ri) 

i = 0 to N 
G = l(pi*Gi) 

30 

f = 0 to N 
B=Z(pi*Bi) 

35 i = 0 to N Equation Group 7 

Note that 

40 pi=ain(1-aj) ® 

j = i+1 to IM 

[0127] In these equations, the term "pi" indicates the composition ratio of each of the pixels of the component "i" to 
the final composite image, I.e., the composite a value of each of the pixels. In this specification, the p value is referred 

45 to as the "contribution degree". 

[0128] According to the present invention, when one of the components is a video image, arithmetic operations are 
perfomied as follow. First, the other components, i.e., the graphical image components are composited using a values. 
Then, the video image component is added to the result of the compositing of the graphical image components. Each 
of the graphical image components is represented by an image file Including a size and a position. Composited on the 

50 OSD plane 1 07, the graphical image components can be efficiently composited with a small amount of memory such 
as the OSD plane 107, i.e., one OSD plane. 

[0129] Accordingly, the processing can be efficiently perfomried with a small amount of memory, I.e., one OSD plane 
for graphical image and one video plane for video image. 

[0130] Instead, if the video and graphical image components are composited in turn in the order of layer, the contri- 
55 butlon degree of each of the pixels for all the components need to be calculated and stored on the memory. This 
requires a large amount of memory and not efficient. 

[0131] Also, the arithmetic operation of the contribution degrees for each of the components includes many multipli- 
cations. More specifically, N*(N-1)/2 multiplications by orders of magnitude are required for N+1 components. 
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[0132] The present Invention solves these problems. More specifically, sequential calculations with a few memories 
and multiplications by N orders are realized. The relationship between the elements and the arithmetic operations are 
as follows. The first compositing unit 108 deals with the addition of the graphical image components, the second com- 
positing unit 109 calculation of the contribution degrees of the video image, and the third compositing unit 110 the 
5 addition of the video image. 

[0133] Fig. 13 shows a program for the arithmetic operations of the present invention written in the fomn of the C 
language. This program focuses on one pixel. The letters "R", "G", and "B" indicate the RGB components of one pixel 
on the OSD plane, and the letter "a" the composite a value of a pixel of a video image corresponding to the pixel. 
Meanwhile, the letters "Ri", "Gi", "Bi", and ^'ai" indicate the RGB components and the a value of a pixel of the "i"th 
10 component corresponding to the pixel, the letters "Rv", "Gv", and "Bv" the RGB components of a pixel of the video 
image corresponding to the pixel. 

[0134] The first to fourth lines Indicate the initialization processing, the fifth to 17th lines the addition of the graphical 
images, and the remaining lines the addition of the video image. Meanwhile, the seventh, eighth, ninth, 12th, 13th, and 
14th Indicate the arithmetic operations perfomned by the first compositing unit 108. Here, in the equations for compos- 
iting the video image, the RGB components are substantially set to be "0", The 10th and 15th lines indicate the arithmetic 
operations performed by the second compositing unit 109 to calculate the composite a value of the video image. The 
fifth, sixth, 11th, 16th, and 17th lines are commonly perfomried by the first compositing unit 108 and the second com- 
positing unit 1 09. Meanwhile, the 1 8th to 20th lines indicate the arithmetic operations performed by the third compositing 
unit 110. Here, the 18th, 19th, and 20th lines may be performed by different threads in parallel. Also, the 18th to 20th 
lines may be an endless loop. 

[0135] Fig. 14 shows the programs that realize the program shown in Fig. 13. The program in Fig. 1 3 is divided into 
programs 1 and 2. The output in the program 2 is an endless loop. The programs 1 and 2 can be perfomned In parallel. 
In this case, the state and process of overlaying can be displayed. 

<The Second Embodlment> 

<Structure> 

[0136] Fig. 15 is a block diagram showing the structure of an image compositing apparatus according to the second 
embodiment of the present invention. 

[0137] In Fig. 15, an Image compositing apparatus 200 includes the input unit 1 01 , a controller 1 500, the EPG gen- 
erating unit 103, the Image holding unit 1 04, the video reproducing unit 105, the video plane 106, an OSD plane 1501, 
a fourth compositing unit 1502, a fifth compositing unit 1503, and the output unit 111 . Note that the elements that have 
the same reference numbers as of the image compositing apparatus 100 have the same functions. The following 
explanation will focus on the elements that have the different reference numbers from the image compositing apparatus 
100. 

[0138] The controller 1500 controls all the elements of the image compositing apparatus 200. More specifically, when 
the user instnjcts to turn the image compositing apparatus 200 ON, the controller 1 500 has the video reproducing unit 
105 reproduce a program. When the user instructs to switch channels, the controller 102 has the video reproducing 
unit 105 reproduce another program. When the user instructs to turn the EPG display ON, the controller controls the 
EPG generating unit 103: the video reproducing unit 105, the fourth compositing unit 1502, the fifth compositing unit 
1503, and the like so as to generate a composite image. 

[01 39] The OSD plane 1501 is composed of a memory or the like. The OSD plane 1 501 holds the RGB components 
and the a values that has been output from the fourth compositing unit 1502. The OSD plane 1501 holds the RGB 
components and the a values corresponding to the pixels of image. 

[0140] When receiving instructions from the controller 1500, the fourth compositing unit 1502 composites the image 
data of the plurality of image files held In the image holding unit 104. In the end, the fourth compositing unit 1502 
generates fourth composite image data and stores the fourth composite image data on the OSD plane 1501. This 
processing is referred to as the "fourth compositing processing" in this specification. 
[0141] Fig. 1 6 is a flowchart showing the fourth compositing processing. 

[0142] To begin with, the fourth compositing unit 1502 sets ail the areas for RGB component on the OSD plane 1501 
and for a value on the video plane 1 06 to be "0" (steps SI 600 and SI 601 ). 

[0143] Next, the fourth compositing unit 1502 repeats the operations at steps SI 602 to SI 61 2. More specifically, the 
fourth compositing unit 1502 composites graphical image components in turn in the bottom-to-top order. Also, the 
fourth compositing unit 1 502 blends a values together for the video image component and stores the blended a values 
on the video plane 1 06. 

[0144] At step SI 602, the fourth compositing unit 1502 reads the image information "i" from the index file in the image 
holding unit 104. Here, the letter "1" is a variable that indcates a number assigned to a component. Numbers are 
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assigned to the overlaid components from the bottom to the top in the ascending numeric order. The image information 
and the image file corresponding to a component "i" are referred to as the "image information "i"" and the "image file 
"i"". In this flowchart, the initial value of "i" is set to "0" and the increment is "1". 

[0145] Then, the fourth compositing unit 1502 reads from the image holding unit 104 the image file "i" that has been 
5 stored in the storage position indicated by the image information "i" (step SI 603). 

[0146] The fourth compositing unit 1502 reads the size and the position to determine an area on the OSD plane 107 
where the component "i" overlaps another component (step SI 604). 

[0147] The fourth compositing unit 1502 judges whether the image type indicated by the image infomiation "i" is 
video image or graphical image (step SI 605). 
10 [0148] When the image type is video image, the fourth compositing unit 1502 copies the a values of the image file 
"i" on the video plane 106 (step S1611). 

[0149] On the other hand, when the imago type is graphical image, the fourth compositing unit 1502 judges whether 
all the a values held on the video plane 106 are '0" (step SI 606). By doing so, the following processing is different 
according to whather a video image is included at the lower layer than the graphical image. 
*5 [0150] When i[ Is judged that all the a values held on the video plane 106 are "0" at step SI 606, the fourth compositing 
unit 1502 copies the a values of the image file "i" on the OSD plane 1501 (step SI 609). 

[0151] On the other hand, when it Is judged that the a values held on the video plane 106 are not all "0", i e., when 
at least one a value Is not "0" on the video plane 1 06, at step SI 606, the fourth compositing unit 1502 calculates new 
a values according to the equation given below using the a values held on the video plane 106 and the a values of 
20 the image file "i". Also, the fourth compositing unit 1502 stores the newly obtained a values on the OSD plane 1501 
(step SI 607). 

aosd (X, y) = ~ ^'.^f/^^ . . — - Equation Group 8 

^ " 1-av(x,y)*(1-ai(x,y)) ^ ^ 

25 

[0152] In this equation, the term aosd(x,y) on the left side indicates an a value that is to be newly obtained. On the 
right side, the term ai(x,y) indicates an a value of the image file "I" and the term av(x,y) an a value held on the video 
plane 106. 

[0153] Moreover, the fourth compositing unit 1502 calculates a values for the video image according to the equation 
30 given below. The calculated a values are stored in the area for a value on the video plane 1 06 (step SI 608). 

av(x,y) = av(x.y)*(1 -ai(x,y)) Equation Group 9 

35 [0154] In this equation, the term av(x,y) on the left side indicates an a value that is to be newly stored on the video 
plane 106. On the right side, the term av(x,y) Indicates an a value that has been stored on the video plane 1 06 before 
the storage of the newly obtained a value and the term ai(x,y) indicates an a value of the image file "i". 
[0155] The fourth compositing unit 1502 perfonns the a blending algorithm for the RGB components of the image 
file "i" and the RGB components in the overlapping area on the OSD plane 1 07. The result of the a blending algorithm 

40 is stored on the OSD plane 1501 (step SI 610). The a blending algorithm Is given below. 

R(x,y) = aosd(x,y)*Ri(x,y)+(1-aosd(x,y))*R(x,y) 
45 G(>^'y) = aosd(x,y)*Gi{x,y)+(1 -aosd(x,y))*G(x,y) 

B(x,y)= aosd(x,y)*Bi(x,y)+(1-aosd(x,y))*B(x,y) Equation Group 10 

[0156] In these equations, the tenns R(x,y), G(x,y), and B(x,y) on the left sides indicate RGB components that are 
50 to be newly obtained. On the right sides, the terms R (x,y), G (x,y), B(x,y), and aosd indicate RGB components and 
an a value held on the OSD plane 1501 . Also, the terms Ri(x,y), Gi(x,y), and Bi(x,y) on the right sides indicate RGB 
components of the image file "1". 

[0157] The fourth compositing unit 1502 performs the processing from steps SI 602 to SI 612 as has been described 
for all the' imago files to complete the fourth compositing processing (step S1612). 
55 [0158] As a resultof the fourth compositing processing the result of the compositing the graphical image components 
is held on the OSD plane 1 501 , the composite a values of the video image component in the area for a value on the 
video plane 106. 

[0159] When receiving nomnal reproduction Instructions from the controller 1500, the fifth compositing unit 1503 
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outputs to the output unit 111 the RGB components that have been held on the video plane 1 06. 

[0160] Also, when receiving the EPG display instructions from the controller 1500, the fifth compositing unit 1503 

performs "fifth compositing processing" to composite the RGB components and a values of the video image data stored 
on the video plane 1 06 with the RGB components of the fourth composite Image data stored on the OSD plane 1 501 . 
5 Then, the fifth compositing unit 1503 outputs the compositing result to the output unit 111. 
[0161] The fifth compositing processing is indicated by the equations given below. 

R(x,y) = a(x,y)*Rv(x,y)+Ro(x,y) 

10 

G(x,y) = a(x,y)*Gv(x,y)+Go(x,y) 

B(x,y) = a(x,y)*Bv(x,y)+Bo(x,y) Equation Group 1 1 

15 [0162] Here, the terms R(x,y), G(x,y), and B(x,y) indicate the RGB components of each of the pixels that are to be 
output to the output unit 111 as a result of the fifth compositing processing. The terms a(x,y), Rv(x,y), Gv(x,y), and Bv 
(x,y) indicate each of the composite a values and each set of the' RGB components of the video image data that is 
stored on the video plane 1 06. IVIeanwhIle, terms Ro(x,y), Go(x,y), and Bo(x,y) indicate each set of the RGB components 
of the fourth composite image data. 

20 [0163] Accordingly, the fifth compositing unit 1503 composites a frame of video images with the fourth composite 
image data whenever the video image frame is updated. 

<Operations> 

25 [0164] Here, an explanation of the operations by the image compositing apparatus 200 will be given below. 

[0165] Fig. 1 7 is a chart for explaining the flow of the operations by the image compositing apparatus 200 at the time 
of EPG display. In Fig. 1 7, operations by elements of the image compositing apparatus 200 are Indicated in the rec- 
tangles and the data flow by the arrows. Note that at the steps in the flowchart in Fig. 17 that have the same reference 
numbers as in the flowchart in Fig. 11 , the same operations as in Fig. 11 are performed. 

30 [0166] When received by the input unit 1 01 (step SI 001 ), the EPG display instructions is sent to the other elements 
via the controller 1500. When receiving the EPG instructions, the EPG generating unit 1 03 generates image files and 
an index file according to the EPG information that has been broadcast. The generated image files and index file are 
stored in the image holding unit 1 04. Also, the EPG generating unit 1 03 infonms the video reproducing unit 1 05 of the 
position and size of the video image component that has been obtained when generating the image files (step SI 002). 

35 [0167] The fourth compositing unit 1 502 combines the RGB components in the overlapping areas in graphical image 
components stored in the image holding unit 1 04. The combined RGB components are stored on the OSD plane 1501 . 
Also, the fourth compositing unit 1 502 calculates the composite a values for the video image and stores the calculated 
composite a values on the video plane 106 (step 81 701). 

[0168] The video reproducing unit 105 reproduces the video image and stores the video image data on the video 
40 plane 1 06 according to the layout indicated by the size and position that have been obtained from the EPG generating 
unit 103 (step S1005). 

[0169] The fifth compositing unit 1 503 weighted the RGB components of the video image component with the com- 
posite a values Then, the RGB components of the fourth composite image data are added to the weighted RGB 
components to obtain the RGB components of the composite image. The obtained RGB components of the composite 
45 image are output to the output unit 111 (step S1702). 

[0170] The output unit 111 displays the composite image RGB components from the fifth compositing unit 1 503 (step 
S1007). 

[0171] In Fig. 17, operations at steps S1702, 1005, and 1701 are performed in parallel. 

50 <Supplemental Remarks> 

[0172] In the compositing method according to the present embodiment, the order of any adjacent two components 
in the component layer are switched when composited as shown in Fig, 18. In this case, the a values of the switched 
two components are updated according to the equations given below. 
55 [0173] In the following equations, each of the a value of the lower component "i" after the switch Is Indicated by the 
term ai+1' and each of the a values of the upper component "i+1" the term ai'. 
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ai+1 = rjz r—TT 

1-ai(1-ai+1) 

ai' = ai (1-ai+1) Equation Group 12 

[0174] Here, Equation Group 12 is obtained as follows. 

[01 75] Suppose that the a values of the components "i" and "i+1 " before the switch are ai and ai+1 , the contribution 
degrees pi and [it+1 of pixels of the components "I" and "i+1 " before the switch to the final composite image are obtained 
according to the equations given below. 

pi=ain(1-aj) ® 
j = i+1 to N 
pi+1 = ai+rn(1-aj) © 

j = i+2 to N Equation Group 1 3 

20 [0176] On the other hand, the contribution degrees pi' and pi+V of pixels of the components "i" and "i+1" after the 
switch to the final composite image are obtained according to the equations given below. 



10 



15 



25 



30 



35 



40 



pi+V = ai+r*(1-ai')n(1-aj) @ 
j - i+2 to N 
pi • = ai'*n(1-aj) ® 

j = i+2 to N Equation Group 14 

[0177] In order to have the same compositing result before and afterthe switch, It is enough that the results of the 

equations® and® are the same and the equations® and® also the same. More specifically, when the following 
equations are solved for the a values ai' and ai+1 , the results of the equations in Equation Group 1 2. 

ain (1-aj) = ai'*ll(1-aj) 
j = i+1 to N j = 1+2 to N 

ai+rn(1-aj) = ai+r*(1-ai') n(1-aj) 

j = i+2 to N j = i+2 to N Equation Group 1 5 

45 [0178] By switching the video image component and a graphical image component using this law so that the video 
image component is positioned at the uppermost as shown in Fig. 19, ail the graphical image components are com- 
posited together and then the video image component is composited with the composite graphical Image components. 
[0179] As shown In the flowchart in Fig. 16, the fourth compositing unit 1502 efficiently calculates the a values nec- 
essary to switch the adjacent components. Also, the fourth compositing unit 1502 efficiently composites graphical 

50 image components and stores the compositing result on the OSD plane 1601. Meanwhile, the fifth compositing unit 
1503 composites the OSD plane 1501 with the video plane 106. 

<The Third Embodinrient> 

55 <Structure> 

[0180] Fig. 20 Is a block diagram showing the structure of an image compositing apparatus according to the third 
embodiment of the present invention. 
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[0181] In Fig. 20, an image compositing apparatus 300 includes the input unit 1 01 , a controller 2000, an EPG gen- 
erating unit 2003, the image holding unit 104, a video reproducing unit 2001 , a video plane 2002, the OSD plane 107, 
a sixth compositing unit 2004, a seventh compositing unit 2005, an eighth comp.oslting unit 2006, and the output unit 
111 . With this structure, the Image compositing apparatus 300 composites a plurality of video image components with 

5 a plurality of graphical image components. Note that the elements that have the same reference numbers as of the 
image compositing apparatuses 100 and 200 have the same functions. The following explanation will focus on the 
elements that have the different reference numbers from the image compositing apparatuses 100 and 200. 
[0182] The controller 2000 controls all the elements of the image compositing apparatus 300. More specifically, when 
the user instructs to turn the image compositing apparatus 300 ON, the controller 2000 has the video reproducing unit 

10 2001 reproduce a program. When the user instructs to switch channels, the controller 2000 has the video reproducing 
unit 2001 reproduce another program. When the user instructs to turn the EPG display ON, the controller 2000 controls 
the EPG generating unit 2003, the video reproducing unit 2001 , the sixth compositing unit 2004, the seventh compos- 
iting unit 2005, the eighth compositing unit 2006, and the like so as to generate a composite image. 
[0183] The video reproducing unit 2001 includes a plurality of reproducing units, i.e., a first reproducing unit, a second 

15 reproducing unit, an Nth reproducing unit. Each of the reproducing units receives a different program. Also, the first 
to Nth reproducing units decode the received programs, reproduce the video images at dozens of frames per second, 
and store the video image data on the video plane 2002 while updating the video image data in frame units. When the 
video image data is stored on the video plane 2002, the video reproducing unit 2001 determines the layout of the video 
images on the final composite image according to the positions and sizes that have been input by the EPG generating 

20 unit 2003. Also, the video reproducing unit 2001 stores the RGB components of the video images on the corresponding 
areas on the video plane 2002. 

[0184] The video plane 2002 includes a plurality of planes, i.e., a first plane, a second plane, and an Nth plane. 
Each of the planes is composed of a memory or the like. Each one of the planes corresponds to a different one of the 
plurality of reproducing units in the video reproducing unit 2001 . The planes hold the video image data from the repre- 
ss ducing units and the composite a values from the seventh compositing unit 2005. 

[0185] The EPG generating unit 2003 obtains the EPG information for the EPG display connposed of a plurality of 
video Image components and graphical image components from the broadcasting station. Also, the EPG generating 
unit 2003 generates a plurality of image files and one index file according to the EPG information. The image files and 
the index file are stored in the image holding unit 104. Furthermore, the EPG generating unit 2003 extracts from the 
30 EPG information the sizes and positions of the plurality of video image components that are to be composited into the 
EPG display image. The extracted sizes and positions are output to the video reproducing unit 2001 . 
[0186] The EPG generating unit 2003 is almost the same as the EPG generating unit 1 03. The difference is that the 
EPG generating unit 2003 is capable of dealing with a plurality of video image components. 

[0187] The sixth compositing unit 2004 composites the image data of the plurality of image files held In the image 
35 holding unit 104. Then, the sixth compositing unit 2004 generates sixth composite image data and stores the sixth 

composite image data on the OSD plane 107. This processing is referred to as the "sixth compositing processing" in 

this specification. The sixth compositing processing is almost the same as the first compositing processing in the first 

embodiment shown in Fig. 9. The difference is that the operation at step S806 is repeated a plurality of times in the 

sixth compositing processing so as to deal with a plurality of video images. 
40 [0188] When receiving instructions from the controller 2000, the seventh compositing unit 2005 calculates composite 

a values for each of the plurality of video image components. Also, the seventh compositing unit 2005 stores the 

calculated a values for each of the video image components on a different corresponding planes. This processing is 

referred to as the "seventh compositing processing" in this specification. 

[0189] Fig. 21 is a flowchart showing the seventh compositing processing. 
45 [0190] To begin with, the seventh compositing unit 2005 initializes the composite a values on the video plane 2002 

(step S21 00). More specifically, the areas for a values in the planes are set to be "0" on the video plane 2002. 

[0191] Next, the seventh compositing unit 2005 repeats the operations at steps S2101 to S2107. 

[01 92] At step S21 01 , the seventh compositing unit 2005 reads image information "i" from the index file in the image 

holding unit 104. Here, the letter "i" is a variable that indicates a number assigned to a component. Numbers are 
50 assigned to the overlaid components from the bottom to the top in the ascending numeric order. The image information 

and the image file corresponding to a component "i" are referred to as the "image Information "i"" and the "image file 

"i"". In this flowchart, the Initial value of "i" is set to "0" and the increment is "1". 

[0193] Then, the seventh compositing unit 2005 reads from the image holding unit 104 the image file "i" that has 
been stored in the storage position indicated by the image information "i" (step S2102). 
55 [0194] The seventh compositing unit 2005 reads the size and the position of the Image file "i" to determine an area 
on the video plane 2002 where the component "i" overlaps another component (step S21 03). 

[0195] The seventh compositing unit 2005 judges whether the image type indicated by the image information "i" is 
video image or graphical image (step S21 04). 
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[0196] When the Image type is video image, the seventh compositing unit 2005 copies the a values of the Image file 
"i" In the area for a value on the "k"th plane on the video plane 2002 (step S21 05). The equation for the operation at 
step S21C5 is given below. 

5 

pk(x.y) = ai(x,y) Equation Group 16 

[0197] In this ejquation, the term ai(x,y) on the left side indicates a value that Is to be stored in the area for a value 
on the "k"th plane on the video plane 2002. The temn ai(x,y) indicates an ot value of the image file "i". The value of "k" 
10 is "1" to "N" and is incremented by "1" whenever the operation at step S2105 is performed. 

[0198] After that, the seventh compositing unit 2005 updates the value of pm(x,y) for the area for a value on the 
"m"th plane, which is different from the "k"th plane, on the video plane 2002 using the equation given below. 



15 



35 



50 



pm(x,y) = (1-al(x,y))*pm(x,y) Equation Group 17 



[0199] Here, the term pnn(x,y) on the left side Indicates each of the values that are to be stored in the area for a value 
on the "m"th plane. The value of "m" is from "1" to "N" excluding "k". 

[0200] Accordi ngly, the arithmetic operations performed at step S21 05 and S21 06 is the same as arithmetic operation 
20 0 in Equation Group 7. As a result of the arithmetic operations at step S21 05 or S21 06, the composition ratio of each 
of the pixels of the video image components to the final composite image, i.e., each of the composite a values, is 
calculated. 

[0201] When the seventh compositing unit 2005 has perfomned the processing for all the image files as has been 
described, the seventh compositing processing is completed (step S2107). As a result, the composite a values for 
25 each of the video image components are stored on the corresponding plane on the video plane 2002. 

[0202] When receiving the EPG display instructions from the controller 2000, the eighth compositing unit 2006 per- 
fomns "eighth compositing processing" to composite the RGB components on the planes on the video plane 2002 with 
the RGB components on the OSD plane 107. Then, the eighth compositing unit 2006 outputs the composite image to 
the output unit 111 . 

30 [0203] When the number of the video image components is "N", the eighth compositing processing Is indicated by 
the equations given below. 



R(x,y) = p1(x,y)*Rvi(x,y)+p2(x.y)*Rv2(x,y)+ ... +pN(x,y)*RvN(x,y)+Ro(x,y) 
G(x,y) = pi (x,y)*Gv1(x,y)+p2(x,y)*Gv2(x,y)-i- .. +pN (x, y)*GvN (x, y) +Go (x, y) 
B(x,y) = P1 (x,y)*Bv1 (x,y)+p2(x,y)*Bv2(x,y)+ ... 
40 +PN(x,y)*BvN(x,y)+Bo(x,y) Equation Group 18 

[0204] Here, thetenns R(x.y), G(x,y), and B(x,y) indicate RGB components of each of the pixels that are to be output 

to the output unit 111 . The terms P1(x,y), Rv1(x,y), Gvl (x,y), and Bv1(x,y) indicate each of the composite a values and 
each set of the RGB components of the video Image data that is stored on the first plane on the video plane 2002. 
45 Accordingly, the terms p2(x,y), Rv2(x,y), Gv2(x.y), and Bv2(x,y) indicate each of the composite a values and each set 
of the RGB components of the video image data that is stored on the second plane on the video plane 2002, and the 
terms pN (x,y), RvN(x,y), GvN(x,y), and BvN(x,y) each of the composite a values and each set of the RGB components 
of the video image data that is stored on the Nth plane on the video plane 2002. Meanwhile, the temis Ro(x,y), Go(x, 
y), and Bo(x,y) indicate each set of the RGB components stored on the OSD plane 107. 



<Operations> 



[0205] The operations by the image compositing apparatus 300, which has the structure that has been described, 
are almost the same as the operations by the image compositing apparatus 1 00 in Fig. 11 . 
55 [0206] The operations by the image compositing apparatus 300 are different from those by the image compositing 
apparatus 100 in the following points. 

[0207] The operation at step SI 002 Is performed by the EPG generating unit 2003. More specifically, the EPG gen- 
erating unit 2003 Infomris the video reproducing unit 2001 of the positions and sizes of not one video image component 
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but a plurality of video image components. 

[0208] Also, the operation at step S1 003 is performed by the sixth compositing unit 2004. 

[0209] The operation at step S1004 is performed by the seventh compositing'unit 2005. More specifically, the seventh 
compositing unit 2005 calculates the composite a values tor each of the plurality of video image components and stores 
5 the calculated composite a values on the video plane 2002. 

[021 0] Furthermore, the operation at step S1 005 is performed by the video reproducing unit 2001 . More specifically, 
the video reproducing unit 2001 reproduces a plurality of video images and stores the reproduced video images on 
the video plane 2002. 

[0211] The operation at step S1006 is performed by the eighth compositing unit 2006. More specifically, the eighth 
10 compositing unit 2006 adds the RGB components of the sixth composite image data to the RGB components of the 
plurality of video image components that have been weighted with thecomposite avalues. Then, the eighth compositing 
unit 2006 outputs the RGB components of the composite image, which have been obtained by the addition, to the 
output unit 111 (step SI 006). 

[0212] The eighth compositing unit 2006 continues to composite the planes on the video plane 2002 with the OSD 
^5 plane 1 07 and output the compositing result as has been described until stop instructions. The operations by the eighth 
compositing unit 2006 are performed in parallel with the operations by the video reproducing unit 2001 , the sixth com- 
positing unit 2004, and the seventh compositing unit 2005. 

<The Fourth Embodiment> 

20 

<Structure> 

[0213] Fig. 22 is a block diagram showing the structure of an image compositing apparatus according to the fourth 
embodiment of the present invention. 

[0214] In Fig. 22, an image compositing apparatus 400 includes the input unit 101, a controller 41 02, an EPG gen- 
erating unit 4103, an image holding unit 4104, the video reproducing unit 105, the video plane 106, the OSD plane 
107, a ninth compositing unit 4108, a tenth compositing unit 4109, the third compositing unit 110, and the output unit 
111 . The compositing apparatus 400 composites Images according to the Porter-Duff rules. The Porter-Duff rules are 
described in T Porter and T. Duff, "Compositing Digital Images", SIGGRAPH 84, 253-259, for instance. In this speci- 
fication, the Porter-Duff rules will be described using Figs. 3A and 3B as necessary. 
[0215] Here, an overall explanation of the Porter-Duff rules will be given below. 

[0216] According to the Porter-Duff rules, there are 12 compositing operations that can be used to combine source 
pixels and destination pixels. The source and destination pixels each has RGB components and an a value. Note that 
the a value is differently defined from the a value in the first to the third embodiments. The a value used In the first to 
third embodiments is defined between two images. In contrast, the a value used in the Porter-Duff rules in the present 
embodiment is an a value defined for each Image. In the Porter-Duff rules an image which Is the result of compositing 
two images which each have an a value also has an a value. When the image which is the result of compositing is 
actually displayed on the screen, a value, which is the RGB value and the a value multiplied together, Is output. 
[0217] The image compositing apparatus 400 of the present embodiment can use eight of the 1 2 compositing oper- 
ations to composite Images. The eight compositing operations are "CLEAR", "SRC", "SRC_OVER", "DST_OVER", 
"SRC_IN", "DST.IN", "SRC_OUT", and "DST_OUT". 

[0218] In the "CLEAR" operation, both of the RGB components and the a values of the destination are cleared. 
Neither the source nor the destination is used as input. In the present embodiment, the destination corresponds to the 
image held on the OSD plane 1 07, while the source to the image of the image file "i". 
[0219] In the "SRC" operation, the source is copied to the destination. 
[0220] In the "SRC_OVER" operation, the source is composited over the destination. 

[0221] In the "DST_OVER" operation, the destination is composited over the source and the result replaces the 
destination. 

[0222] In the "SRCJN" operation, the part of the source lying inside of the destination replaces the destination. 
[0223] In the "DSTJN" operation, the part of the destination lying inside of the source replaces the destination. 
[0224] In the "SRC_OUT" operation, the part of the source lying outside of the destination replaces the destination. 
[0225] In the "DST_OUT" operation, the part of the destination lying outside of the source replaces the destination. 
[0226] Here, an explanation of the elements different from the other embodiments will be given. 
[0227] The controller 41 02 controls all the elements of the image compositing apparatus 400. More specifically, when 
the user instnjcts to turn the image compositing apparatus 400 ON, the controller 41 02 has the video reproducing unit 
105 reproduce a program. When the user instructs to switch channels, the controller 41 02 has the video reproducing 
unit 1 05 reproduce another program. When the user instructs to turn the EPG display ON, the controller 41 02 controls 
the EPG generating unit 41 03, the video reproducing unit 1 05, the ninth compositing unit 41 08, the tenth compositing 



18 

BNSDOCID: <EP 1 170942A1J_> 



EP1 170 942 A1 



unit 4109, the third compositing unit 110, and the tike so as to generate a composite image. 

[0228] As in the case of the first embodiment, the EPG generating unit 4103 obtains the EPG information from the 
broadcasting station. Also, the EPG generating unit 41 03 generates a plurality of image files and one index file according 
to the EPG information. The image files and the Index file are stored in the image holding unit 4104. Furthermore, the 
5 EPG generating unit 41 03 extracts from the EPG information the size and position of the video image component that 
is to be composited into the EPG display image. The extracted size and position are output to the video reproducing 
unit 105. 

[0229] The image holding unit 41 04 holds a plurality of image files and one index file. 

[0230] The index file, which is used for controlling the plurality of image files, includes a plurality of pieces of image 

10 information. Each piece of image information corresponds to a different image file. The plurality of pieces of image 
information are arranged in the index file in the same order as the overlaying order of components. Each piece of image 
infomnation includes an image type, an operation typo, and a storage position. The image type indicates whether the 
image of the corresponding image file is a graphical image or a video image. The operation type indicates which one 
of the 12 composition operations in the Porter-Duff rules is used. The storage position indicates the position of the 

15 beginning of the corresponding image file in the image holding unit 4104. 

[0231] Fig. 23A shows an example of index fiJe. Fig. 23B shows the correspondence between the number and the 
operation type. An index file 4410 in Fig. 23A includes image information 4421 , 4422, and 4423. The image information 
4422 is overlaid on the image information 4421 , and the image information 4423 is overlaid on the image information 
4422. IVIore specifically, the image information 4421 corresponds to a component that is positioned at the lowest layer. 

20 the image information 4422 corresponds to a component at the next layer, and the image information 4423 corresponds 
to a component at the uppemiost layer. The values in a column 411 indicate the image types of the components. More 
specifically, the \/a\ue "0" indicates the graphical image, while the value "1" the video image. The values in a column 
41 3 indicate the operation types, More specifically, the value "1 " in the line 4422^ column 413 indicates the "SRC_OVER" 
operation, and tie value "2" in the line 4423, column 413 and in the line 4421, column 413 the "SRC" operation. The 

25 values in a column 412 indicates the storage positions. 

[0232] Meanwhile, images 430, 440, and 450 correspond to the components 202, 203, and 204, respectively. The 
images 430, 440 and 450 show the image data in the corresponding image files in the form of image. Here, the image 
440 shows that all the RGB components of the image data corresponding to the component 203 are "0". According to 
the Porter-Duff rules, a values are set for all the images. For this reason, any a values are set for the image 430 even 

30 though the image 430 corresponds to the component at the lowest layer. 

[0233] The image holding unit 41 04 holds the plurality of image files and the index file. 

[0234] When receiving instructions from the controller 41 02, the ninth compositing unit 4108 composites the image 
data of the plurality of Image files held In the Image holding unit 4104 to generate ninth composite image data. The 
ninth composite image data are stored on the OSD plane 1 07. This processing is referred to as the "ninth compositing 
35 processing" in this specification. 

[0235] Fig. 24 is a flowchart showing the ninth compositing processing. 

[0236] In the flowchart in Fig. 24, the same operations as in Fig. 9 are perfomried at the steps that have the same 
reference numbers as in the flowchart in Fig. 9. The following explanation will be focused on the operations at the steps 
with different reference numbers. 
40 [0237] At step S4802, the ninth compositing unit 41 08 reads the image file "i" stored in the storage position Indicated 
by the image information "i" and the operation type indicates by the image infonnation "i". 

[0238] At step S4805, the Porter-Duff rules shown in Fig. 25 is perfomried In pixel units according to the operation 
type read at step S4802. 

[0239] At step S4806, the RGB components of the video image are sets as "0" and an operation shown in Fig. 26 
45 is performed in pixel unfits according to the operation type. 

[0240] At step 4807, an a value calculation as shown in Fig. 27 is performed in pixel units according to the operation 
type. 

[0241] Note that eight among 12 compositing operations of the Porter-Duff rules are shown in Figs. 25, 26, and 27. 
[0242] When receiving instructions from the controller 41 02, the tenth compositing unit 41 09 generates the a values 
50 of the video plane 1 06 to the OSD plane 1 07 from the image data stored in the image holding unit 41 04. The generated 
a values are stored on the video plane 106. 

[0243] Fig. 28 is a flowchart showing the compositing processing. 

[0244] In the flowchart in Fig. 28, the same operations as in Fig. 10 are performed at the steps that have the same 
reference numbers as in the flowchart in Fig. 10. The following explanation will be focused on the operations at the 
55 steps with reference numbers different from the flowchart in Fig. 10. 

[0245] At step S1202, the tenth compositing unit 41 09 reads the image file "i" indicated by the image information "i" 
and the operation type indicates by the image Information "i". 

[0246] At step S4905, the tenth compositing unit 41 09 performs an operation shown in Fig. 29 in pixel units according 
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to the operation type that has been read at step S1202. A graphical image is overlaid on a video image in principle, 
so that the components of the video image are weakened. In Fig. 29, the value "a" indicates the a value of each of the 
pixels on the video plane 1 06, and the value "ai" indicates the a value of each of the pixels of the read image file that 
are overlaid on the video plane 1 06. Theoretically, the a value is "0" to "1 ". Actually, however, the a value is represented 

5 by a value "0" to "255" or "0" to "15". 

[0247] At step S4906, the tenth compositing unit4109 perfomris an operation shown in Fig. 30 in pixel units according 
to the operation type. Note that the ninth compositing unit 41 08 operates in synchronization with the tenth compositing 
unit 4109. At steps S1202, S903, S904, and S907, the results at steps S4802, S803, S804, and S807 may be used. 
Also, the operations at steps SI 202, S903, S904, and S907 may be performed in synchronization with the operations 

10 at steps S4B02, S803, S804, and S807. Meanwhile, the operations at steps S4905 and 4906 need to be completed 
before the operation at step S807. 

[0248] Accordingly, the image compositing apparatus 400 composites images according to the Porter-Duff rules with 
a small amount of memory and smaller number of multiplications as in the case of the first embodiment. 
[0249] Fig. 31 shows a program for the arithmetic operations in the fourth embodiment of the present invention written 
15 In the form of the C language. This program focuses on one pixel. The letters "R", "G", "B", and "a" indicate the RGB 
components and the a value of one pixel on the OSD plane, and the letters "av " indicate the a value of a corresponding 
pixel on the video plane. 

[0250] The first to fifth lines indicate the initialization processing. The eighth, 11th, and 14th indicate operations 
perfomied by the ninth compositing unit 4108. Meanwhile, the ninth and 12th lines indicate arithmetic operations per- 

20 formed by the tenth compositing unit 41 09. In the ninth and 1 2th lines, the contribution degree (the p value), i.e. , the 
composite a value, is calculated. The 16th, 17th, and 18th lines indicate the arithmetic operations by the third com- 
positing unit 110. Here, the 16th, 17th, and 18th lines may be performed by different threads in parallel. Also, the 18th 
to 20th lines may be an endless loop. More specifically, the arithmetic operation in the eighth line is shown in Fig. 26, 
and the arithmetic operation in the 11th line in Fig. 25. Meanwhile, the arithmetic operation in the ninth line is shown 

25 in Fig. 30, and the arithmetic operation in the 12th In Fig. 29. Here, the letter "a" In Fig. 29 corresponds to the letters 
"av" in Fig. 31 . More specifically, the arithmetic operation In the 14th line is shown in Fig. 27. 

[0251] Accordingly, the image compositing apparatus 400 in the fourth embodiment composites a plurality of graph- 
ical images and calculates the composition ratio of a video image according to the operation type of the Porter-Duff 
rules designated for each of the images. Then, the image compositing apparatus 400 composites each of the frames 

30 of the video image with the composite graphical Image according to the operation type. By doing so, the plurality of 
graphical images need not to be composited with a frame of the video image whenever the video image frame is 
updated. As a result, the calculation is performed under light load and the processing speed is increased. As a result 
of the higher processing speed, images are composited and composite image is displayed in real time according to 
the reproduction rate of the video image. Accordingly, every time one frame is developed in the frame buffer, the frame 

35 is composited with composite graphical image due to the real time Image composition. As a result, the capacity of the 
frame buffer for developing video image frames can be decreased. This Is effective. 

[0252] While an explanation of the image compositing apparatus according to the present invention has been given, 
other possible modifications will be given below. 

40 (1 ) In the first and second embodiments, the composite a values of the video Image component are stored on the 

video plane 106. When the number of video Image component Is one, however, the composite a values of the 
video image component may be stored on the OSD planes 107 and 1 501 . 

(2) In the first to fourth embodiments, the pixel data are stored in the memory and the like in the order of R, G, B, 
and a. The pixel data, however, can be stored in a different order, for instance, the order of a, R, G, and B, or a, 

45 B, G, and R. Also, the components R, G, B, and a can be stored in component units instead of pixel units. 

(3) The data length of each of thecomponents R, G, B, and a Is one byte. Accordingly, the RGB components each 
have gradation values out of 256 gradation values, i.e., "0" to "255". In total, the RGB components express about 
16,700,000 colors. The data length, however, can be four bits, i.e., so-called, "high color". 

(4) In the first to fourth embodiments, YUV components can be used instead of RGB components. 

50 (5) The Image holding units 1 04 and 4104 hold pixel data as the Image data of the graphical Image components 

in the preferred embodiments. Instead, the image holding units 104 and 4104 can hold vector data, i.e., data 
composed of expressions and graphics such as straight line, dot, and circle to develop the vector data into pixel 
data when the first compositing unit 1 08 composites image data together. By doing so, the memory capacity can 
be decreased. 

55 (6) The operational procedure of the Image compositing apparatus 100, 200, 300, and 400 can be a computer 

program executed by a general-purpose computer or an apparatus having a function of executing a prograni. 

Especially, a computer program corresponding to the image compositing apparatus 400 is effective for elimi- 
nating' the need for eight kinds of operational circuit corresponding to the eight kinds of operation of the Porter- 
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Duff rules. Also, the computer program is effective for having one processor that executes the computer program 
carry out all the eight kinds of operation. 

Furthermore, the computer program can be distributed by being recorded on a recording medium or through 
a variety of communication channels and the like. The recording medium includes the IC card, the optical disc, 
5 the flexible disk, the ROM, and the like. 

(7) The third compositing unit 110 is constructed to perform the aformenlioned addition for each frame in video 
reproductioi, in other words to perfomri addition in synchronization with playback of frames, however, the third 
compositincj unit 110 noes not have to be synchronous, but may be asynchronous. The same applies to the fifth 
compositincj unit 1503 and the eighth compositing unit 2006. 

10 

industrial Applicability 

[0253] The present invention relates to an image display apparatus that composites and outputs a plurality of images 
together, and can be used particularly in a television that receives digital broadcasts. 

15 

Claims 

1. An image compositing apparatus that composites images together into a final composite image, the images in- 
20 eluding a video image and a plurality of graphical images, the image compositing apparatus comprising: 

first obtaining means that obtains the graphical images and composition information including an image com- 
positing order the composition information being used for calculating a composition ratio of each of the images 
to the final composite image; 

25 first compositing means that composites the graphical Images to generate one composite graphical image 

according to the composition information; 

calcula:ing means that calculates a composition ratio of the video image to the final composite image according 

to the composition information; 

second obtaining means that obtains frames of the video image; and 
50 second compositing means that composites one of the obtained frames at a time with the composite graphical 

image using the composition ratio of the video image. 

2. The image compositing apparatus of Claim 1 , wherein the composition Infomnation further Includes: 

55 coefficients, each of which corresponds to a different one of the images; and 

arithmetic information which has blending algorithms which each correspond to the images, the blending al- 
gorithrrs using the coefficients. 

3. The image compositing apparatus of Claim 2, further comprising a first frame buffer that stores images, and a 
40 second frame buffer that stores frames of the video image, 

wherein 

the first compositing means reads the graphical images obtained by the first obtaining means, according to 
the image compositing order, composites each of the read graphical images with a storage content of the first 
frame buffer using the coefficients and the arithmetic information, and replaces the storage content of the first 
buffer with a result of compositing one of the read graphical images with the storage content, 
the second obtaining means stores the obtained frames In the second buffer, and 

the second compositing means composites each of the frames stored in the second buffer with an image in 
the first frame buffer using the composition ratio of the video image. 

The Image somposlting apparatus of Claim 3, wherein 

the first compositing means perfonns a blending algorithm on the image In the first frame buffer using a coef- 
ficient and arithmetic information corresponding to the video image after compositing a graphical image im- 
mediately preceding the video image In the compositing order and before compositing a graphical image Im- 
mediately succeeding the video image with the storage content, and replaces the content of the first frame 
buffer with a result of the blending algorithm. 
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5. The image compositing apparatus of Claim 3, further comprising a display screen, wherein 

the first compositing means generates the composite graphical image, the second obtaining means obtains 
the frames, and the second compositing means composites the frames with the composite graphical image 
5 in parallel with each other. 

6. The image compositing apparatus of Claim 1 , wherein 

the composition information further includes a composition coefficient for each of the images, each composition 
10 coefficient indicating a composition ratio of one of the images to a composite of the image with at least another 

one of the images. 

7. The image compositing apparatus of Claim 6, further comprising a first frame buffer that stores images, and a 
second frame buffer that stores frames of the video image, 

15 wherein 

the first compositing means reads the graphical images according to the Image compositing order obtained 
by the first obtaining means, composites each of the read graphical images with a storage content of the first 
frame buffer using the composition coefficients, and replaces the storage content of the first buffer with a result 
^0 of compositing one of the read graphical images with the storage content, 

the second obtaining means stores the obtained frames in the second buffer, and 

the second compositing means composites each of the frames stored in the second buffer with an image in 
the first frame buffer using the composition ratio of the video image. 

25 8. The Image compositing apparatus of Claim 7, wherein 

the first compositing means performs a blending algorithm on the image in the first frame buffer using a com- 
position coefficient corresponding to the video image after compositing a graphical image immediately pre- 
ceding the video image in the compositing order and before compositing a graphical image immediately suc- 
30 ceeding the video Image with the storage content, and replaces the content of the first frame buffer with a 

result of the blending algorithm. 

9. The image compositing apparatus of Claim 7, further comprising a display screen, wherein 

35 the first compositing means generates the composite graphical image, the second obtaining means obtains 

the frames, and the second compositing means composites the frames with the composite graphical image 
in parallel with each other. 

10. The image compositing apparatus of Claim 6, wherein 

40 ... - 

the image compositing order indicates a first overlaying order of the images, 

the composition coefficients area values, each of which indicates a composition ratio of one of the images to 
a composite of the image with images below the image in the first overlaying order, and 
the calculating means calculates the composition ratio of the video Image to the final composite Image from 
45 a values of the video image and images above the video image in the first overlaying order 

11 . The Image compositing apparatus of Claim 10, further comprising a first frame buffer that stores images, and a 
second frame buffer that stores frames of the video image, 

wherein 

50 

the first compositing means reads the graphical images, starting from a lowest layer according to the image 
compositing order obtained by the first obtaining means, composites each of the read graphical images with 
a storage content of the first frame buffer using the a values, and replaces the storage content of the first 
buffer with a result of compositing one of the read graphical images with the storage content; 
55 the second obtaining means stores the obtained frames in the second buffer, and 

the second compositing means composites each of the frames stored in the second buffer with an image in 
the first frame buffer using the composition ratio of the video image. 



22 

BNSDOCID: <EP 1 170942A1_I_> 



EP1 170 942 A1 



12. The image compositing apparatus of Claim 11 , wherein the first compositing means perfonns a blending algorithm 
on the image in the first frame buffer using an a value corresponding to the video Image after compositing a 
graphical Image Immediately preceding the video image In the compositing order and before compositing a graph- 
ical image immediately succeeding the video image with the storage content, and replaces the content of the first 

5 frame buffer with a result of the blending algorithm. 

13. The image compositing apparatus of Claim 11 , further comprising a display screen, wherein 

the first compositing means generates the composite graphical image, the second obtaining means obtains 
the frames, and the second compositing means composites the frames with the composite graphical Image 
in paral el with each other 

14. The image compositing apparatus of Claim 10, further comprising: 

switchir g means for switching two adjacent images in the first overlaying order to rearrange the images in a 
second overlaying order; and 

updating means for updating a values of the switched two adjacent images so that results of compositing in 
the first and second overlaying orders are the same, wherein 

the first compositing means generates the composite graphical Image, the calculating means calculates the 
composition ratio of the video Image to the final composite image, and the second compositing means com- 
posites the frames with the composite graphical image using the second overlaying order and the updated a 
values. 
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15. The image compositing apparatus of Claim 14, wherein 

the two adjacent images are Images "i" and "i+1 " that are each "i"th and "i+1 "th images from a bottom of the 
first overlaying order, and 

the updating means sets a values of the images "i+1" and "i" as a[i]*(1-a[i+1]) and a[l+1]/(1-ot[l]*(1-a[i+1]), 

respectively. 

16. The image compositing apparatus of Claim 1 , further comprising a storage unit for storing a plurality of graphical 
images that are obtained by the first obtaining means, wherein 

the graphical images are each made up of (A) image data which has pixels, a number of which is no larger 
than pi>els of the final composite image and (B) layout infomnation which indicates a layout of the images on 
the fina composite Image, and 

the first compositing means generates the composite graphical image, the calculating means calculates the 
composition ration of the video image to the final composite Image, and the second compositing means com- 
posites the frames with the composite graphical image for an overlapping part of the images that is determined 
40 by the layout information. 

17. The image compositing apparatus of Claim 1 . further comprising a storage unit for storing the plurality of graphical 
Images obtained by the first obtaining means; 

the plurality of graphical images are represented by vector data; and 

the first compositing means generates the composite graphical image after converting the vector data to the 
pixels. 

18. An image compositing apparatus that composites images together into a final composite image, the images in- 
50 eluding a pit rality of video images and a plurality of graphical images, the image compositing apparatus comprising: 

first obtaining means that obtains the graphical images and composition infonnation including an image com- 
positing order, the composition infomnation being used forcalculating a composition ratio of each of the images 
to the final composite image; 

first compositing means that composites the graphical images to generate one composite graphical image 
according to the composition information; 

calculating means that calculates composition ratios of the video images to the final composite image according 
to the composition Information; 
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second obtaining nneans that obtains frames of tine video images one frame at a time; and 

second compositing means that composites, each time one frame is obtained, the obtained frame with the 

composite graphical image using the composition ratios of the video images. 

5 19. An Image compositing apparatus that composites Images 'together Into a final composite image, the Images in- 
cluding a video image and a plurality of graphical images, the image composition apparatus comprising: 

first obtaining means that obtains the graphical images.; 

first compositing means that composites the graphical images to generate one composite graphical image; 
10 second obtaining means that obtains frames of the video image; and 

second compositing means that composites one of the obtained frames at a time with the composite graphical 
image. 

20. A computer- readable recording medium storing a program that has a computer composite images together into a 
15 final composite image, the images including a video image and a plurality of graphical images, the program com- 
prising: 

a first obtaining step for obtaining the graphical images and composition infonnation that includes an image 
compositing order, the composition information being used for calculating a composition ratio of each of the 
20 images to the final composite image; 

a first compositing step for compositing the graphical images to generate one composite graphical image 
according to the composition Information; 

a calculating step for calculating a composition ratio of the video image to the final composite Image according 
to the composition information; 
25 a second obtaining step for obtaining frames of the video image; and 

a second compositing step for compositing one of the obtained frames at a time with the composite graphical 
image using the composition ratio of the video image. 

21. A program that has a computer composite images together into a final composite image, the images including a 
30 video Image and a plurality of graphical images, the program comprising: 

a first obtaining step for obtaining the graphical images and composition Infonnation that includes an Image 
compositing order, the composition information being used for calculating a composition ratio of each of the 
images to the final composite image; 
35 a first compositing step for compositing the graphical images to generate one composite graphical image 

according to the composition Information; 

a calculating step for calculating a composition ratio of the video image to the final composite Image according 

to the composition information; 

a second obtaining step for obtaining frames of the video image; and 
'io a second compositing step for compositing one of the obtained frames at a time with the composite graphical 

image using the composition ratio of the video image. 
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1 R=o ; 

2 G=o ; 

3 B=o ; 

4 a =0 ; 

5 for(i=0 ; i<=N ; i++) { 

6 (VIDEO = component i) { 

7 R = (l-aO*R; 

8 G=(l~ai)*G; 

9 B=(l-ai)*B; 

10 or = tfi ; 

11 I else 1 

12 R= i*Ri+(l- cr i)*R ; 

13 G= ai*Gi + (l- ai)*G \ 

14 B = ai*Bi+(l- ai)*B ; 
15" or = a *(1— a i) \ 

16 t 

17 I 

18 R=R+ a *Rv *, 

19 G=G4- a *Gv ; 

20 B=B+a*Bv; 
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1 R = 0 ; 

2 G=o; 

3 B=0 ; 

4 a =0 ; 

5 for(i=0; i<=N ;i++) i 

6 if (VIDEO = component i) 1 

7 ^ R = (l- «i)*R ; 

8 G = (l- ai)*G ; 

9 B = (l- ai)*B ; 

10 a = « i ; 

11 1 else i 

12 R= ai*Ri + (l- ai)*R ; 

13 G= a i*Gi+(l - a i)*G *, 

14 B= a i*Bi+(l - o i)*B ; 

15 a = a*(l-ai); 

16 I ..' -.-.^.N / .nn- 

17 I 
[program 2] 

1 while(tme) 1 

2 R=R-f- a *Rv ; 

3 G=G+a*Gv; 

4 B=B+a*Bv; 

5 I 
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FIG.24 
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FIG.27 
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FIG.31 



1 R=0 ; 

2 G=o; 

3 B=o ; 

4 a=0; 

5 '«v=o; 

6 for(i=0 ; i<=N ; i++) \ 

7 if (VIDEO = component i) 1 

8 R,G,B update A 

9 • ' a V update A 

10 I else 1 

11 R,G,BupdateB 

12 a V update B 

13 I 

14 a update 

15 1 

16 R=R+ a v*Rv ; 

17 G = G+av*Gv; 

18 B=B+«v*Bv; 
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